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Executive  Summary 


The  Air  Traffic  Control  Specialist  (ATCS)  Performance  Measurement  Database  is  a  compilation 
of  performance  measures  and  measurement  techniques  that  researchers  have  used.  It  may  be 
applicable  to  other  human  factor  research  related  to  air  traffic  control  (ATC).  The  database 
provides  a  valuable  tool  to  assist  evaluations  of  air  traffic  equipment.  Using  standard  database 
techniques,  a  researcher  can  select  measures  appropriate  to  the  experimental  questions  under 
study.  The  database  provides  citations  for  the  primary  sources  from  which  the  measure  was 
obtained  and  additional  references  for  further  information.  Further,  the  authors  have  included  a 
bibliography  of  human  performance  measurement  references  as  an  additional  source  of  reference 
information. 

The  database  represents  an  important  tool  that  can  be  used  in  conjunction  with  ATC  simulators, 
generic  sector  configurations,  scenarios,  and  other  procedures  used  in  assessing  ATC  system 
safety  and  capacity.  Having  a  set  of  measures  with  standardized  parameters  will  increase  the 
reliability  of  results  across  experiments  and  enable  comparisons  of  results  across  evaluations. 
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1.  Introduction 


The  Federal  Aviation  Administration  (FAA)  has  established  strategic  goals  of  improved  Air 
Traffic  Control  (ATC)  system  safety  and  capacity.  Measures  of  ATC  system  performance  and 
Air  Traffic  Control  Specialist  (ATCS)  performance  are  required  to  accurately  determine  which 
elements  of  the  system  need  to  be  changed  to  achieve  those  goals  and  to  determine  when 
progress  has  been  attained.  The  primary  goal  is  to  develop  a  comprehensive  set  of  ATCS 
performance  measures  that  relate  to  ATC  system  safety  and  capacity.  Development  of  this 
database  is  one  of  several  objectives  required  to  achieve  this  goal. 

1.1  Background 

There  are  several  well-known  measures  of  overall  ATC  system  effectiveness  used  in  assessments 
(Hopkin,  1995).  However,  the  task  of  controllers  within  ATC  systems  primarily  involves 
cognitive  activities,  which  are  difficult  to  measure  directly.  Instead,  researchers  must  infer 
evaluations  of  many  aspects  of  ATCS  performance.  Although  a  large  number  of  performance 
measures  have  been  used  in  ATC  evaluations,  their  relationships  to  system  effectiveness  are 
inconsistent  and  not  well  understood.  Whereas  many  of  the  elements  affecting  overall  system 
performance  are  well  documented,  the  relationships  between  controller  performance  and  system 
effectiveness  are  still  in  initial  stages  of  exploration 

ATCS  performance  measures  allow  researchers  to  examine  the  relationship  between  what  the 
controller  does  and  how  well  the  system  works.  Performance  measures  are  useful  for  a  wide 
range  of  activities,  including 

•  mitigation  of  risk; 

•  validation  of  operations  concepts,  operational  requirements,  and  equipment 
specifications; 

•  evaluation  of  ATCS/computer  functional  allocation; 

•  assessment  of  the  effectiveness  of  proposed  procedures  and  ATCS/system  interactions; 

•  development  of  display  design; 

•  identification  of  design  incompatibilities; 

•  evaluation  of  information  displays  and  ATCS  interface  usability; 

•  diagnosis  of  usability  and  effectiveness  issues  to  identify  limiting  factors; 

•  determining  benefits  for  cost-benefits  analyses; 

•  identifying  sources  of  human  error  and  methods  to  reduce  them;  and 

•  selection  and  training  of  personnel. 
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A  primary  goal  of  human  factors  research  in  ATC  is  to  establish  the  link  between  ATCS 
performance  and  system  performance.  One  of  the  objectives  of  the  ongoing  research  activities  at 
the  FAA  William  J.  Hughes  Technieal  Center  has  been  to  identify  this  relationship.  The 
National  Airspace  System  (NAS)  Effectiveness  Model  (Figure  1)  conceptualizes  the  relationships 
between  variables  of  NAS  safety,  efficiency,  capacity,  and  controller  performance. 

This  model  illustrates  how  the  ability  of  controllers  to  adapt  to  changes  in  the  dynamic  ATC 
environment  impacts  system  effectiveness.  For  example,  fluctuations  in  separation  distances 
influence  system  capacity  and  affect  the  taskloads  of  controllers.  ATCSs  must  have  the  ability  to 
allocate  resources  to  adjust  for  changing  system  demands  without  compromising  safety  or 
effectiveness.  The  impact  of  system  effectiveness  on  aviation  operations  is  significant  in  terms 
of  safety,  flight  delays,  and  excess  fuel  usage.  Factors  such  as  the  characteristics  of  the  air 
traffic,  weather,  and  the  air  carrier  operating  procedures  affect  ATC  system  effectiveness. 
However,  human  performance  such  as  the  behavior  of  ATCSs,  airway  facilities  specialists,  and 
supervisors  plays  a  major  role  in  defining  system  effectiveness. 


National  Airsoace  Svstem 

System  Effectiveness  Measures 

Capacity  Delay  Time  Operational  Errors 
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Figure  1.  National  Airspace  System  Effectiveness  Model. 

To  study  ATCS  performanee,  it  is  necessary  to  understand  their  critical  functions,  tasks,  and 
associated  behaviors.  Because  ATC  is,  in  large  part,  a  cognitive  activity,  ATCS  performance 
measures  should  be  relevant  to  the  performance  of  cognitive  tasks.  PERI  Technologies  and  its 
contractors  developed  the  ATCS  Functional  Performance  Model,  shown  in  Figure  2,  through 
review  of  various  ATC  task  analyses  of  operations,  concepts,  current  theories,  and  controller 
input.  The  model  identifies  the  relationships  between  the  controller’s  primary  activities  and 
associated  behaviors.  These  relationships  identify  appropriate  measures  that  assess  the  quality 
and  effectiveness  of  those  activities  and  behaviors. 
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Figure  2.  ATCS  Functional  Performance  Model. 

To  analyze  the  wide  range  of  ATCS  activities,  the  model  divides  the  overall  task  into  groups  of 
related  activities  that  have  a  start  point,  an  identifiable  process,  and  an  endpoint  or  result.  This 
model  classifies  ATCS  functions  into  three  categories  (Situation  Assessment,  Planning  and 
Decision  Making,  and  Implementation)  and  describes  the  behaviors  occurring  in  a  sequential 
fashion.  To  make  an  effective  decision,  the  ATCS  must  have  developed  a  high  level  of  situation 
awareness.  These  variables  influence  system  effectiveness,  and  nearly  all  of  the  controller  tasks 
can  be  classified  under  one  or  more  of  these  major  categories.  Each  functional  category  is 
characterized  by  its  associated  tasks,  and  each  task  involves  observable  behaviors.  This  model  is 
useful  for  categorizing  the  many  accepted  and  potential  performance  measures  that  currently 
exist  in  the  literature. 

Many  tools  are  needed  to  enable  ATC  researchers  to  effectively  apply  this  model  and  evaluate 
these  performance  measures  in  the  context  of  their  research.  The  performance  measurement 
database  is  one  such  tool. 

1.2  Purpose 

There  are  two  important  purposes  for  developing  and  applying  a  performance  measurement 
database.  The  first  is  to  compile  effective  ATCS  performance  measurement  techniques  into  a 
single  source.  The  second  is  to  promote  standardization  of  parameters  across  research  projects 
and,  therefore,  enable  comparisons  of  results  across  evaluations. 
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This  database  will  be  particularly  valuable  for  researchers  with  limited  exposure  to  ATC  research 
methods.  The  authors  assembled  this  database  in  Microsoft  Excel  rather  than  in  a  more  complex 
database  manager  because  of  its  near  universal  availability.  Further,  such  software  allows 
researchers  to  explore  for  measures  appropriate  to  the  experimental  questions  they  are 
addressing. 

1.3  Value  to  ATC 


The  primary  objective  of  performance  measurement  is  to  provide  a  better  understanding  of  NAS 
critical  elements  and  to  help  to  diagnose  and  solve  system  performance  issues.  From  a  human 
factors  research  standpoint,  one  important  question  is  how  to  establish  the  link  between  ATCS 
performance  and  system  effectiveness. 

The  ATCS  performance  measurement  database  is  a  compilation  of  measures  and  measurement 
techniques  that  have  been  proven  effective  for  use  in  human  factor  research  related  to  ATC. 
Figure  3  illustrates  some  of  the  potential  applications  for  this  database.  The  following 
paragraphs  describe  the  elements  in  Figure  3. 


Figure  3.  Database  applications. 

Functional  Prototypes.  Measures  of  performance  and  effectiveness  are  essential  for  cost- 
effective  system  development.  Functional  prototypes  can  provide  data  to  determine  the  potential 
benefits  of  proposed  NAS  functions.  The  cost  of  prototype  evaluation  is  minor  compared  to  the 
cost  of  equipment  development  and  design  changes  late  in  the  process.  Evaluation  of  functional 
performance  can  restrain  the  tendency  to  design  sophisticated,  complicated  software  with  a  heavy 
investment  and  little  demonstrated  benefits. 
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ATC  Simulation  Evaluations.  ATC  simulation  evaluations  are  used  to  examine  the  relationship 
between  proposed  changes  in  equipment,  operating  procedures,  and  ATCS  performance.  These 
comparisons  can  then  be  used  to  identify  potential  problem  areas  before  major  investments  are 
made  in  development  and  implementation.  The  results  of  simulations  provide  a  basis  for 
changes  that  ensure  compatibility  with  the  workforce  and  user  acceptance.  Changes  made  early 
are  much  less  costly  and  less  disruptive  to  the  development  schedule. 

Equipment  Development.  The  equipment  development  process  progresses  through  the  stages  of 
initi^  design  concept,  through  detailed  design,  to  production.  Performance  measurement  is  the 
only  objective  measure  of  progress  during  demonstrations  and  provides  a  basis  for  determining  if 
the  equipment  will  achieve  the  performance  goals.  Performance  measures  that  maintain  a  focus 
on  end-item  performance  relative  to  the  system  goals  help  to  ensure  that  the  final  product  meets 
expectations. 

Operational  Test  &  Evaluation  (OT&E).  Performance  measures  can  provide  data  to  support  or 
refute  the  subjective  evaluations  of  subject  matter  experts.  Performance  measurement  imposes  a 
method  that  ensures  reliability  of  the  results.  Generic  sectors  and  standard  traffic  scenarios  can 
be  used  to  avoid  the  difficulty  of  comparing  data  from  ATCSs  with  different  experience. 
Standardized  procedures  will  help  comparison  of  results  from  different  evaluations.  OT&E  often 
does  not  adequately  address  human  performance  issues.  The  presence  of  a  readily  available 
measurement  set  may  improve  the  situation  in  the  future. 

Field  Operations.  Currently,  we  measure  ATCS  effectiveness  in  terms  of  arrivals  and  departures 
and  the  amount  of  delay  associated  with  those  operations.  These  numbers  are  compared  to  the 
engineered  performance  standards  for  a  particular  airport  under  a  given  weather  condition  and 
runway  configuration.  Operational  errors  are  calculated  in  numbers  per  facility  and  are  used  as 
an  indicator  of  safety  measurements.  Certain  individual  ATCSs  and  teams  of  controllers  are 
more  effective  by  reaching  higher  numbers  of  operations  while  committing  fewer  operational 
errors  than  others.  Field  evaluations  of  system  and  individual  performance  are  limited  under 
Labor  Management  Relation  (LMR)  Agreements  and  are  subjectively  completed  by  supervisory 
personnel  as  being  either  satisfactory  or  not  satisfactory.  Basic  tools  for  performance 
improvement  and  systems  enhancement  are  restrained  by  an  extremely  high  percentage  of 
satisfactory  measurements  in  the  system  and  the  LMR  Agreements  on  over-the-shoulder  and 
tape-talk  evaluations.  Through  performance  issue  studies,  the  Research  Development  and 
Human  Factors  Laboratory  located  at  the  Technical  Center  can  provide  necessary  data  to 
substantiate  the  needed  support  for  change. 

2.  Database 

The  database  contains  performance  measures  that  researchers  have  used  for  assessing  ATCS 
performance.  The  database  and  associated  references  are  included  as  Appendix  A  and  can  also  be 
accessed  and  downloaded  via  the  FAA  William  J  Hughes  Technical  webpage  (www.faa.tc.gov). 
An  additional  source  of  human  performance  measures  are  contained  in  Appendix  B.  At  this  time, 
it  is  unlikely  that  the  database  includes  all  of  the  measures  and  measurement  techniques  that  are 
applicable  to  ATC  assessments.  However,  it  is  intended  to  be  an  adaptive  research  tool,  and  the 
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authors  invite  your  nominations  of  other  measures  for  the  database.  One  of  the  most  important 
features  of  this  database  is  that  any  new,  valid  measures  of  performance  and  measurement 
techniques  can  be  easily  integrated  and  the  database  automatically  updated. 

2.1  Database  Description 

The  layout  of  the  database  and  the  process  for  extracting  information  from  it  appear  in  Figure  4. 
Filtering  the  database  can  be  done  by  searching  keywords,  a  specific  reference,  or  measurement 
type.  Probably  the  most  efficient  method  of  searching  is  by  measurement  type.  As  shown  in 
Figure  4,  the  definitions  for  each  measurement  type  are  located  within  the  database  and  can  be 
easily  accessed.  For  example,  if  a  researcher  is  interested  in  what  performance  measures  are 
associated  with  examining  situation  assessment  in  the  en  route  environment,  he  or  she  can  filter 
the  database  specifically  for  those  items.  The  database  then  produces  a  listing  of  references  of 
previous  studies,  the  performance  measures,  and  the  measurement  techniques  associated  with 
situation  assessment  and  the  en  route  environment.  The  researcher  can  then  decide  on  which 
performance  measure  or  measurement  technique  best  suits  requirements  of  the  current  research 
question 
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Figure  4.  Database  construction  and  features. 
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2,2  Database  Configuration 


The  database  is  stractured  around  four  categories  in  which  ATCS  and  ATC  measures  can  be 
obtained:  ATCS  Performance  Measures,  Measures  of  Air  Traffic  Effectiveness,  the  ATCS 
Functional  Performance  Model,  and  the  applicable  Air  Traffic  Environment.  The  types  of 
measures  included  within  each  category  are  defined  in  the  following  sections. 

2.2. 1  ATCS  Performance  Measures 


Performance  measures  included  in  this  category  describe  how  effective  the  controller,  control 
team,  or  system  is  in  accomplishing  ATC  activities.  Some  major  measures  are  delays  to  traffic 
and  violations  of  separation  rules. 

Conflict:  Violation  of  safe  separation  minima  between  two  aircraft.  In  terminal  airspace,  a 
conflict  occurs  when  the  distance  between  two  aircraft  is  <3  miles  laterally  and  <1000  ft 
vertically.  En  route  conflicts  occur  when  spacing  becomes  <5  miles  laterally  and  <1000  ft 
vertically.  At  altitude  above  Flight  Level  290,  the  minimum  vertical  separation  distance  is  2000 
ft  (FAA,  1998).  There  are  exceptions,  such  as  when  one  pilot  sees  the  aircraft  ahead  and  accepts 
visual  separation,  or  both  aircraft  are  established  on  parallel  localizers. 

Complexity:  Sector  and  traffic  characteristics  that  cumulatively  add  to  create  a  complex  set  of 
rules,  requirements,  and  tasks  for  the  controller  when  controlling  aircraft  in  the  sector.  ATC 
complexity  is  composed  of  sector  and  traffic  complexity  factors  such  as  control  adjustments  (i.e., 
merging,  spacing,  and  speed  changes;  climbing  and  descending  flight  paths;  and  mix  of  aircraft 
types).  However,  the  authors  recognize  that  a  considerable  amount  of  confusion  exists  about  this 
construct  and,  like  human  workload,  there  is  unlikely  to  ever  be  universal  agreement  concerning 
its  meaning. 

Error  (Conflict  and  Non-conflict):  A  conflict  error  (operational  error)  occurs  when  a  failure  of 
equipment,  human,  procedural,  and/or  system  elements,  individually  or  in  combination  results  in 
less  than  the  separation  minima.  Non-conflict  errors  include,  but  are  not  limited  to, 
misidentification  of  information  from  the  radar  display,  acceptance  of  incomplete  position 
information,  and  interpreting  flight  progress  strips  incorrectly. 

Communication:  Typical  ATCS-to-aircraft  communications  involve  using  a  standard 
phraseology  with  aircraft  identification,  destinations,  departure  instructions,  altitude  assignments, 
holding  instructions,  and  flight  plan  modifications.  Communication  between  controllers  includes 
coordination  between  and  within  sectors,  handoffs,  and  pointouts. 
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Taskload:  System  demands  placed  upon  the  controller  by  the  current  situation,  including  air 
traffic  volume,  mix,  complexity  of  routings,  and  weather;  the  number  of  tasks  or  frequency  of 
task  occurrence  associated  with  a  specific  job. 

Workload:  The  effects  of  taskload  on  the  individual  controller  and  the  degree  to  which  he/she 
accepts  it.  Workload  is  influenced  by  the  controller’s  internalized  standards  of  performance, 
ability,  and  experience. 

Other:  ATC  tasks  and  required  procedures  not  specifically  or  exclusively  captured  under  any  of 
the  above  variable  categories  including  the  use  of  J-rings,  history  trails,  and  strip  bay 
management. 

2.2.2  Air  Traffic  Effectiveness 

Safety  measures  include  counts  of  conflicts  or  separation  violations  that  occur,  ratings  of  ATCSs 
and  observers  of  system  safety  (using  notes,  questionnaires,  or  debriefing  after  each  run  or  series 
of  runs),  and  various  measures  and  indices  of  aircraft  proximity  such  as  slant  range  distance  and 
the  aircraft  proximity  index  (Paul,  1990). 

Capacity  is  the  maximum  number  of  aircraft  and  aircraft  procedures  that  can  be  safely  handled  by 
the  ATCS  and  the  equipment  he/she  is  using.  System  capacity  varies  as  a  function  of  a  number 
of  variables  such  as  weather  conditions,  radio  frequency  congestion,  and  sector  size. 

Efficiency  concerns  the  frequency  and  duration  of  delays  along  with  fuel  and  resource 
management.  ATC  system  efficiency  encompasses  accuracy  of  data  entry,  handoffs,  and 
coordination  between  sectors. 

2.2.3  ATCS  Functional  Performance 

Controller  functional  performance  is  a  diagnostic  of  how  a  controller  performs  tasks  as 
distinguished  from  controller  productive  performance.  The  distinction  between  this  category  and 
ATCS  Performance  Measures  is  that,  here,  the  focus  is  on  the  process  rather  than  the  results  or 
product.  The  development  of  the  ATCS  Functional  Performance  Model  (Figure  2)  resulted  in  the 
identification  of  three  behavior  categories  that  can  effectively  classify  all  ATCS  cognitive 
performance:  situation  assessment,  planning/decision  making,  and  plan/decision  implementation. 

In  the  ATC  environment,  situation  assessment  entails  developing  and  maintaining  the  picture. 

For  the  purposes  of  the  database,  situation  assessment  represents  the  following  tasks  as  shown  in 
Figure  5:  a)  acquiring  elements  of  current  situation,  b)  integrating  relevant  elements  of  the 
situation  into  the  picture,  and  c)  evaluating  the  situation  to  identify  critical  events/problems  that 
need  to  be  addressed.  An  ATC  event  can  be  thought  of  as  any  situation  that  needs  attention, 
regardless  of  whether  it  is  actually  a  problem.  For  example,  a  conflict  might  indicate  a  problem 
or  it  might  be  a  routine  sequencing  and  spacing  of  aircraft  onto  the  final  approach.  Situationeil 
assessment  is  considered  a  precursor  to  other  ATCS  behaviors.  Before  any  action  can  be  taken, 
the  controller  must  evaluate  the  situation  to  determine  if  there  is  a  need  for  action.  To  be 
effective,  the  controller  must  have  knowledge  of  the  status  and  dynamics  of  the  individual 
aircraft,  knowledge  of  relevant  procedures,  and  a  comprehension  of  the  total  situation. 
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Figure  5.  ATCS  tasks  in  situation  assessment. 

The  recognition  of  ATC  events  within  situation  assessment  is  the  impetus  for  planning  and 
decision-making  behavior.  Planning  and  decision  making,  as  shown  in  Figure  6,  is  the  process  of 
reviewing  the  situation,  determining  available  options  to  achieve  the  desired  goal,  and  deciding 
which  option  to  implement.  The  ATCS  decides  on  priorities,  aircraft  sequence,  speed,  altitude, 
and  flight  routes  within  the  context  of  the  situation.  This  behavior  results  in  a  decision,  plan, 
solution,  or  strategy.  Usually,  the  resulting  plan  or  decision  requires  an  action  (e.g.,  issuing  a 
clearance  for  a  flight  plan  change  of  heading,  altitude,  or  airspeed).  Implementation  is  the 
ATCS’s  next  step. 


Figure  6.  Planning  and  decision  making  tasks. 
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Implementation  is  the  process  by  which  the  ATCS  acts  on  the  previously  determined  decisions. 
Implementation  behaviors  (Figure  7)  include  communication  and  coordination,  issuing 
clearances,  and  assessing  the  progress  of  the  plan.  Progress  assessment,  or  compliance 
monitoring,  simply  means  directing  part  of  subsequent  situation  assessment  behavior  towards  a 
targeted  search  for  information  to  evaluate  the  success  of  the  implementation.  This  is 
represented  in  the  model  by  a  feedback  loop.  The  execution  of  a  decision  affects  the  situation 
and,  therefore,  the  situation  must  be  continuously  updated  and  evaluated. 


Figure  7.  Implementation  tasks. 
2.2.4  Air  Traffic  Environments 


The  NAS  includes  three  types  of  environments:  En  Route,  TRACON,  and  Tower  (Nolan,  1994). 
Although  it  is  generally  considered  part  of  the  en  route  environment.  Oceanic  is  included  in  the 
database  as  a  fourth  air  traffic  environment.  The  different  airspace  categories  have  distinct 
characteristics. 

En  Route:  En  route  ATC  service  provided  on  Instrument  Flight  Rules  (IFR)  flight  plans  when 
aircraft  are  operating  between  departure  and  destination  terminal  areas. 

TRACON:  A  terminal  ATC  facility  associated  with  an  ATC  tower  that  uses  radar  to  provide 
approach  control  services  to  aircraft. 

Tower:  A  terminal  facility  that  uses  air/ground  communications,  visual  signaling,  and  other 
devices  to  provide  ATC  services  to  aircraft  operating  in  the  vicinity  of  an  airport  or  on  the 
movement  area.  The  tower  authorizes  aircraft  to  land  or  takeoff  at  the  airport  controlled  by  the 
tower  or  to  transit  the  Class  D  airspace  area  regardless  of  flight  plan  or  weather  conditions  (IFR 
or  Visual  Flight  Rules  [VFRs]). 

Oceanic:  Operating  procedures,  track  structure,  and  separation  standards  vary  across  different  air 
control  regions.  Minimum  separations  allowed  over  the  ocean  are  much  larger  than  in  the 
domestic  airspace  due  to  lack  of  aircraft  surveillance  and  inefficient  High  Frequency 
communications.  Therefore,  oceanic  airspace  capacity  is  limited. 
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3.  Conclusion 


Reliable  information  about  the  performance  and  effectiveness  of  ATCSs  and  how  their 
performance  affects  the  system  is  essential  to  understanding  system  design,  selection,  training, 
and  operational  concepts  and  procedures.  Development  and  testing  of  controller  performance 
measures  are  part  of  an  ongoing  process,  and  the  database  has  been  designed  to  allow  easy 
integration  of  the  most  current  ATC  research  findings.  Researchers  are  encouraged  to  include 
newly  discovered  measures  of  controller  performance  and  to  operationally  define  the  existing 
measures  within  the  database.  This  will  increase  the  reliability  of  results  and  will  foster  the 
development  of  standardized  parameters  so  that  valid  comparisons  between  experiments  can  be 
made. 

Clearly,  the  ATCS  is  a  vital  element  of  ATC  system  operations,  but  there  is  a  gap  in 
understanding  the  impact  of  changes  in  controller  performance  on  system  effectiveness.  New, 
valid  measures  of  controller  performance  are  needed  to  understand  factors  that  improve  or 
degrade  performance.  A  solid  understanding  of  those  performance  factors  is  particularly 
important  to  evaluate  the  impact  of  the  various  automation  concepts  in  ATC  system  design  that 
are  being  proposed. 
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Appendix  A 

Performance  Measurement  Database 
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System  Effectiveness  SEM  The  SEM  set  measures  many  different  factors 

Measures  associated  with  the  safety  and  efficiency  of  the 

system:  conflictlon,  occupancy,  communication, 
and  delay 


Assessment  Battery  Department  of  Defense  laboratory,  (2)  proven 

validity,  (3)  relevance  and  (4)  sensitivity  to 
hostile  environments  and  sustained  operations 
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systems. 
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Multidimensional  scaling  MDS  Multidimensional  scaling  was  used  for  direct 

and  indirect  reconstnjction  of  cognitive  maps  as 
well  as  a  diagnostic  version  of  the  methodology 
for  studying  mental  rotation  of  three- 
dimensional  objects 
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Radio  frequency  None  This  adds  to  complexity  due  to  the  increased 

congestion  difficulty  in  communicating  with  a  large  number 

of  aircraft  on  the  same  radio  frequency 
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